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COORDINATED AUDIO AND VISUAL OMNIDIRECTIONAL RECORDING 



FIELD OF THE INVENTION 

This invention relates to simultaneous recording of panoramic visual data and panoramic 
audio dat and the application of the directed audio (or radiated) signal to focus the visual 
recording on the source of the directed audio signal, 
BACKGROUND AND INFORMATION DISCLOSURE: 

The use of video photography has evolved steadily over the past forty years in its 
application to surveillance situations and environments. Applications include stationary 
systems such as are found in commercial establishments, and on moving bases such as 
robots, on submarines, vehicles, etc. 

For example, U.S, Patent 6,016,385 to Yee et al discloses a real time remotely controlled 
robot having a head with a pair of television cameras (eyes) and a pair of microphones 
(ears) and a pair of arms mounted on a mobile trunk, an operated located at a command 
center receives audio and visual signals from the robot and, in response, issued commands 
that control movement of the trunk and arms to perform tasks, 

U, S, Patent 4,453,085 to Pryor discloses an electro-optical system for monitoring the 
positioning of a movable arm. 
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U. S. Patent 4,604,559 To Hawkes et al discloses a robot controlled by a user wherein an 
audio signal is fed back from the robot to the user regarding force and movement 
parameters of the robot operation. 

U. S. Patent 5,299,288 to Glassman et al discloses an image directed robot system for 
surgical applications including an optical tracking camera supplying data to the controlling 
processor. 

One of the major developments in video photography in recent years has been the 
panoramic camera. According to this technology a lens with a 360^ field of view stores 
the entire field in memory and various areas are selected and projected according to the 
interests of the viewer. This technique has many applications, for example, in real estate 
advertisements where features of the location are selected and enlarged for detailed 
examination by the viewer. 

U.S. Patent 5,920,337 to Glassman et al discloses an omnidkectional visual image detector 
and processor (incorporation herein by reference). There is described a reflective round 
lens (either a cone or spherical section) which reflects the object beam from a surrounding 
view (encircling the lens) to a direction parallel to the axis of the rotund lens. The beam 
then passes through an objective lens and is focussed onto a CCD (charge coupled device) 
where image data is detected and processed for storage or image presentation on a monitor. 
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In the context of this specification, the term, "visually exaniine" means that, a narrow field 
of view is selected covering a few degrees from an entire 360^ field of view recorded by a 
panoramic video camera. 

SUMMARY OF THE INVENTION. 

It is an object of this invention to develop a surveillance device that enables a user to 
visually examine a location that has a source of a signal. The signal is an audial signal , in 
one version or an electr omagnetic signal (radio, microwave, light, infrared, etc) in another 
version. . 

This invention is directed toward an omnidirectional camera having a lens for viewing a 
360^ field of view; a visual memory for storing visual data at addresses representing the 
field of view; an omnidirectional detector for detecting the direction of location of signal 
source, A display selection circuit is provided for selectiag visual data from an address in 
the visual memory corresponding to the direction of location. The visual data is displayed 
on a system monitor. 

In one version, a directional audio detector and a panoramic CCD (charge coupled device) 
camera installed in the center of a site, (e.g., store, prison yard, battlefield, etc.) The camera 
is continually generating visual data of a 360^ view of the site. If a sound is generated 
from a location in the site, ( an intruder, a shout, a shot, etc.) then the audio receiver of this 
invention wiU detect the sound and its direction relative to the camera and direct the camera 
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to display that portion of the stored data representing the visual image of the source. 

Another object of the invention is to provide a panoramic audio detector that is capable of 
receiving an audio signal from a source located any where on a 36(P circumference around 
the audio detector, "point" to the location of the source of sound and then amplify the audio 
signal for more detailed examination of the sound. The data corresponding to the 
direction of the source of sound is used to control the display selected from the visual data 
stored by the camera. 

In one embodiment, the panoramic audio detector of this invention comprises a plurality of 
audio detectors, strung one after the other on a circle. Each audio detector comprises a 
piezo electric element, a flip-flop, and coupling circuitry between the piezo electric element 
and its respective flip-flop. The coupling circuitry enables the piezo element to set its 
respective flip-flop at the instant a sound impulse is received by the piezo-electric. The 
direction of the source of sound will determine which flip-flop is set before all the other 
flip-flops. The first flip-flop to be set will store its address (referred to herein as the 
"source address") in a central register and the central register will ignore signals from all 
other flip flops during the sampUng period. 

The register will thereby have stored an address corresponding to the direction of the 
source. This address is used to select camera data from the panoramic camera for 
locaUzed display of the source of sound. 
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The phase of the sound signal reaching the detector closest to the source will be ahead of 
the phases of the same sound signal reaching the other (more distal) detectors. The 
invention includes an audio "focusing" circuit that adjusts the phase of the signal from each 
detector so that each signal from the respective detector is in phase with the detector nearest 
the source of sound. An adder adds together all of the phase adjusted signals to generate a 
"summation sound" signal from the source. 

In another embodiment, a focusing circuit coupled to the panoramic camera displays the 
image of a location in response to motion in that location. The panoramic viewing system 
has two video memories, and a comparator that is configured to compare point by point 
image data at each memory address of one visual memory to data at the corresponding 
addresses of the other visual memory. A difference between a signal stored at one address 
in one memory and the signal at the same address in the other memory indicates motion at 
the location and this difference causes the circuit to display the image of the location where 
motion has occxirred. . 

BRIEF DESCRIPTION OF THE FIGURES: 

Fig. 1 shows an audial omnidirectional detector of this invention. 

Fig. 2A and 2B show one embodiment of a selection means. 

Fig. 3 is a timing diagram of the emboeiment of fig. 4. 

Fig. 4 shows another emboeiment of a selection technique. 

Fig, 5 Show an audial array having two detectors. 

Fig. 6 shows a version of the invention being a panoramic teleconference table. 
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DESCRIPTION OF ILLUSTRATIVE EXAMPLES: 



Turning to a discussion of the drawings, fig. 1 is a schematic diagram of one version of the 
invention 10 in which the directional function of the panoramic camera responds to a sound 
coming from a source at a location on the circumferential field of view. 

There is shown an omnidirectional visual imaging system 13 including an array of "sub " 
lenses 12, focussing lens 14 and charge coupled device array 16. The omnidirectional 
visual imaging system 13 is of the type disclosed by U.S.Patent 5,920,337 to Classman 
which is incorporated herein by reference. 

An audio detector system 11 is shown as a string of detectors 30 mounted on a circle 
that is coaxial with an axis 18 of the lens 12. The detectors 39 are preferably piezo electric, 
capacitive, semiconductor elements or similar pressure sensitive devices well known in the 
art. . 

Each detector 30 is coupled to a selection circuit 49 that responds to the first detector to 
receive a sound wave radiated from a source 34 by shutting off the signals from all of the 
other detectors. 

Sound generated by a source 34 from a location on the circumferential field of view 
reaches first the detector 30 A closest to the source 34 before it reaches the other detectors 
thereby shutting off the signals from aU of the other detectors. An address register 36 
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registers the address of the closest detector 30A and ignores the signals from the remaining 
audial detectors 32, The registered address is applied to display control circuit 38. The 
visual display circuit 38 selects the visual data from the corresponding address in the video 
memory 40 and displays an image of the location on the system monitor 42. 

Also displayed is a number indicating the azimuthal direction of the source of sound 
relative to the camera. 

One embodiment of a selection circuit 49 is shown in fig. 2.A. showing an array (ring) of 
audial detectors 30 . Fig, 2B shows a portion of circuit 49 to better advantage than fig. 2A. 
The output terminal of each audial detector 30 is connected to an input terminal of an AND 
gate 51. Each detector 30 has its own AND gate 51. 

The output terminals of all AND gates 51 have two connections: 

(i) One connection 63 is to an address register (not shown) which activates a portion of the 
CCD to send an image signal representing that portion of the field of view to a video 
monitor. 

(ii) The second connection of all output terminals 53 of all the AND gates 51 is connected 
to a bus 57. Bus 57 is connected to an input terminal of a single inverter 55 whose output 
teiminal is connected to bus 59. Bus 59 is connected to a second input terminal 61 of all 
AND gates 51. 

Initially there is no sound incident on any of the detectors so that the input signal to all of 
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the gates 51 is zero* When a sound signal is radiated from a sound source, the closest 
detector is the first to receive the signal and sends a positive signal to one input of its gate 
Since the signal on the second input terminal is zero (not negative) the first gate to receive 
the signal from its detector 30 will send a signal to the address register 63. The positive 
signal from the first gate to receive the sound signal is inverted by inverter 55 and the 
negative signal is applied through the second bus 59 to the second input terminal of all the 
remaining gates thereby turning off all remaining gates. 

The video memory 40 reads the selected address from register 63 and applies the image 
data on monitor 42. 

An alternate scheme for detecting the direction of the sound is shown in fig, 4. There is 
shown a ring of audial detectors 30 arranged relative to the omnidirectional lens. The lens 
is not shown in fig.4 but is arranged coaxially with the audial detector array as discussed 
with regard to fig. 1. 

Each detector 30 of the array of detectors forms a pair of detectors with one member of 
each pair being positioned on a circumferential boundary oppositely from another 
member of the pair. For example, detectors 30A and 30B form one pair, detectors 30C 
and 30D form another pair A plurality of difference circuits 37 is provided although only 
two difference circuits are shown in fig. 4. a difference circuit for each pair of detectors, 
one pair of detectors for each difference circuit 37. 

Fig. 4 shows further: a inverters 39 (two are shown), 
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one inverter for each pair of detectors, one pair of detectors for each inverter; 

a rectifiers 49, one rectifier 49 for each pair of detectors 30, one pair of detectors 30 for 
each rectifier 49; 

an address register 41 having a plurality of address input terminals, one address terminal 
for one difference circuit and one difference circuit for one address terminal; 

one member of each pair of detectors 30 is connected to transmit an image signal to an 
input terminal of one of the inverters 39; 

an output terminal of the inverter is connected to an input terminal of one of the difference 
circuits; 

another one of each pair of detectors is connected to another input terminal of its difference 
circuit; 

each one of the rectifiers 49 has an input tenninal connected to an output terminal of the 
difference circuit 37; 

each one of the rectifiers 49 has an output terminal connected to one of the input terminals 
of the address register 41 . 
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the address register has an output terminal configured to emit an address signal responsive 
to a null signal from one of the rectifiers. 

The timing diagrams of figs. 3.1- 3.7 show the signals generated by a sound pulse 46 
incident on the array of detectors 30. Detector 30C is the "leading" detector where the 
phase shift of the incoming wave will be assumed to be zero as shown in fig. 3,4. Figs. 
3.1 and 3.2 show that the signals received by detectors 30A and 30B have shifted an equal 
amount, 0, so that the difference between their signals (fig. 3,3) is zero. The inverted 
signal, Fig. 3.5, from detector 30D has shifted by 20 thereby giving pulse 3.6. 

Fig. 3.7 is a plot of the peak output of each difference circuit 37 vs. 0. Interpolation of the 
plot to Zero yields the value of 0 being the dkection of the sound 46. 

When no sound is incoming, the output S3 = 0 (fig. 3.7) . When a sound wave 46 is 
incident on the array 30, then a finite pulse 3.7 is generated, from the respective one of the 
difference circuits that stores that the signal in the register at its azimuthal address 0 
Selection circuit 45 then applies the address Q to video memory 46 to locate the video data 
corresponding to the source of the sound 46. Video memory 46 is connected to CCD 
array 16 for storing an image signal generated by the CCD 36 responsive to the visual 
image of the surrounding field formed by omnidirectional lens 13. The selected image data 
is presented on video monitor 42. Not shown in fig. 2 but shown in fig. 1, the axis of the 
audial array is comcident with the axis of the omnidirectional lens 13. 

Fig. 5 shows another embodiment of the invention being a system which relies on the 
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phase difference of the sound wave between a pair of audial detectors 50 and 52. to present 
on a video monitor 42 an image of an area of a field of view formed by an omnidirectional 
lens 13 on a CCD array and stored in video memory 40. 

If P represents the period between the arrival of the earliest detectable portion of a sound 
signal, (such as a shot) to each detector. The velocity of sound is C. Then the angle, 0, 
of the direction of the sound relative to the audio detector line is 

COS0 = I^/d 

In practice, visual data is stored in the visual memory 40 at addresses corresponding to 
the angle 0. The phase detector 51 measures the time period P being the difference in 
times of arrival of the signal from source S to first audial detector 50 and detector 352. 
The value of 0, calculated by the formula, gives the address of the image data for 
generating a local image to be displayed directly on the monitor 42. 

Several embodiments have been described of a surveillance system being a combination of 
audial detectors that sense the du'ection of an incoming sound wave and a visual system 
that presents an image of the area includes the source of the sound. 

Other variations and modification of this invention may be contemplated after reading the 
specification and studying the drawings that are within the scope of the invention. 

For example other forms of lens (other than an omnidirectional lens may be used to collect 
the image data. 
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Fig. 6 shows an embodiment for application in televising a conference. The conferees 62 
(seven are shown)are located (seated) at locations around a panoramic camera 64 
positioned on a support 65 (e.g., table). Each location has a microphone 66. Each 
location is viewed through a corresponding section of the lens of camera 64 When the 
voice of the conferee 62 nearest a particular microphone 64 at the conferee's location is 
detected by the microphone 64, the voice signal is amplified and transmitted to signal 
selection controller 70 which conditions the panoramic camera 64 to project the image 
formed by the corresponding section 68 on the monitor 72. The signal selection controller 
70 is shown in phantom in fig. 6 because it is preferably located under the table 65. 

In another arrangement, the signal selection means includes includes an array of buttons 
(switches) on a console 73. Each switch, when closed, connects a selected one of the 
microphones 62 to a source of power (not shown) enabling the selected microphone 62 to 
convert sound to an electrical signal for transmission to the monitor, coincident with 
selection of the corresponding section of the field of view for viewing on the monitor. In 
this arrangement, selection of the microscope occurs simultaneously with the selection of 
the video signal for viewing corresponding to the selected microphone. 

In other versions: 

the system is mountable on a cart or robot and transportable into an otherwise inaccessible 
environment. 
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the principles of the invention are adaptable to underwater surveillance. 

the system may be adapted to use at night by using a visual system that is sensitive to 
infrared 

The audial detector in various embodiments include piezoelectric detectors, capacitive 
detectors and semiconductor detectors such as are well known in the art. 

I therefore wish to define the scope of the invention by the appended claims. 
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What is claimed is: 

L A surveillance system which comprises: 

a camera means for generating an image signal of a field of view; 

an audial means for detecting the direction of a source of sound located in said field of 
view; relative to a location of said camera; 

signal selecting means arranged in operable combination with said audial means and said 
camera means for selecting a portion of said image signal generated from an area 
containing said source of sound; 

monitor coupled to said signal selecting means for presenting an image of said area 
containing said source of sound. 

1. The surveillance system of claim 1 wherein said camera means comprises: 

an unidirectional lens for forming an image of a panoramic field of view; 

a CCD array operably arranged to receive said image and generate an image signal; 
for transmittal to said monitor subject to control by said signal selecting means. 



-14- 



3, The surveillance system of claim 1 wherein said camera means comprises: 

a unidirectional lens for forming an image of a panoramic field of view around said 
camera; 

a CCD array arranged to receive said image and generate an image signal; 

video memory means coupled to said CCD array for storing said image signal; 

said video memory coupled to said signal selection means for transmitting said portion to 
said monitor, 

4, The surveillance system of claim 1 wherein said camera means comprises: 

an omnidirectional lens for forming an image of a 360^ field of view; 

a CCD array arranged to receive said image and generate an image signal 

for transmittal to said monitor subject to control by said signal selecting means. 

5, The surveillance system of claim 1 wherein said camera means comprises: 

an omnidirectional lens for forming an image of a 360^ field of view around said camera; 
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a CCD array arranged to receive said image and generate an image signal; 

video memory means coupled to said CCD array for storing said image signal; 

said video memory coupled to said signal selection means for transmitting said portion to 
said monitor. 

6. The system of claim 5 wherein said audial array comprises: 

a plurality of audial detectors circumferentially arranged around said omnidirectional lens: 

said audial array having a central axis coincident with said omnidirectional lens; 

circuit means for generating a signal representing an address in said video memory 
corresponding to said direction of said source of sound relative to said camera.and 
applying image data stored at said address to said monitor for display of said area including 
said sound source; 

7, The system of claim 6 wherein said circuit means comprises: 

a plurality of AND gates, each gate connected to only one of said plurality of detectors, 
each detector coupled to only one of said AND gates to transmit an image signal from each 
said detector to one input terminal of said respective AND gate; 
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a first bus connected to an output terminal of each AND gate: 
an inverter having an input terminal connected to said first bus; 

a second bus connected to an output temiinal of said inverter and to a second input terminal 
of said AND gate; 

an address register connected in operable;e combination with said plurality of AND gates 
for storing in said address register an address of one of said AND gates in response to an 
image signal received from said AND gate; 

said address expressed as an angular position of said detector in said array of detectors; and 

said selection means connected to said register comprising means for selecting from said 
video memory said portion of said image signal generated from an area containing said 
source of sound and having an address stored in said register whereby a source of sound 
generates a sound wave that is incident on a nearest detector enabling said selection circuit 
to read said address of said detector. 

8» The system of claim 6 wherein said circuit means comprises: 

each detector of said array of detectors forming a pair of detectors with one member of said 
pak being positioned on a circumferential boundary oppositely from another member of 
said pair; 
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a plurality of difference circuits, difference circuit for each pair of detectors, one pair of 
detectors for each difference circuit; 

a plurality of inverters, one inverter for each pair of detectors, one pair of detectors for each 
inverter; 

a plurality of rectifiers, one rectifier for each pair of detectors, one pair of detectors for each 
rectifier, 

an address register having a plurality of address input terminals, one address terminal for 
one difference circuit and one difference circuit for one address terminal; 

one member of said pair of detectors connected to transmit an image signal to an input 
terminal of one of said plurality of inverters; 

an output terminal of said inverter connected to an input terminal of one of said plurality 
of difference circuits; 

another one of each pair of detectors connected to another input terminal of its difference 
circuit; 

one of said plurality of rectifiers having an input terminal connected to an output terminal 
of said difference circuit; 
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each one of said rectifiers having an output terminal connected to one of said input 
terminals of said address register, 

said address register having an output terminal configured to emit an address signal 
responsive to a null signal from one of said rectifiers* 

9,* The surveillance system of claim 5 wherein said omnidirectional audial detector 
comprises: 

two detectors spaced a distance, D, from one another and having identical detection 
sensitivities; 

said signal arriving at one of said detectors at one instant and arriving a period of time, P, 
later at said second detector, 

means for measuring the period of time, P; 

means for calculating a direction angle, 0, being the angle which the direction from the 
audial detector to the source makes with the line connecting the two detectors using the 
relation: 

COS0 = ^*^/d, 
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said display selection means applying the angle, 0, to identify the address in visual 
memory of the visual data. 

10. The surveillance system of claim 1 arranged to display a meeting of a group of 
conferees, which comprises: 

a support for said camera means; 

each conferee positionable at a respective one of a group of locations in said field of view 
of said camera means and each conferee being a source of sound from said respective 
location; 

said audial means comprising a group of microphones, each positioned at a respective one 
of said locations and corresponding to said direction, being one of a group of directions of 
said conferees relative to said camera means; 

said signal selecting means arranged in operable combination with said group of 
microphones and said camera means for selecting a portion of said image signal generated 
from one of said locations in response to sound generated at said location by one of said 
conferees at said location for presentation on said monitor. 

11. The surveillance system of claim 1 arranged to display a meeting of a group of 
conferees, which comprises: 
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a support for said camera means; 

each conferee positionable at a respective one of a group of locations in a field of view of 
said camera means and each conferee being a source of sound from said respective 
location; 

said audial means comprising a group of microphones, each positioned at a respective one 
of said locations and corresponding to said direction, being one of a group of directions of 
said conferees relative to said camera means; 

said signal selecting means arranged in operable combination with said group of 
microphones and said camera means for selecting a portion of said image signal generated 
from one of said locations for presentation on said monitor. 

12. The surveillance system of claim 1 1 wherein said signal selecting means comprises: 

a group of switches, one of said switches connecting a power means to one of said 
microphones respectively whereby said power means enables said microphone to convert 
an audio signal to an electrical signal for transmission to said signal selecting means and to 
select said portion of said image signal corresponding to said location where said 
microphone is located for presentation on said monitor when said respective switch is 
closed* 

said group of switches accessible to an operator enabling said operator to select any one of 
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said sections for transmitting an image and sound 

13. . The surveillance system of claim 10 wherein said camera means comprises: 

an unidirectional lens for forming an image of a panoramic field of view; 

a CCD array operably arranged to receive said image and generate an image signal; 
for transmittal to said monitor subject to control by said signal selecting means. 

14. The surveillance system of claim 10 wherein said support means comprises a table. 

15.. The surveillance system of claim 1 1 wherein said support means comprises a table. 

16. Amethodformonitoringaconferenceof a group of conferees which comprises: 

positioning said group of conferees at respective locations around a video camera means 
having an nomnidirectional field of view providing that each conferee is in said field of 
view of said camera means wherein said camera means has a video signal control for 
projecting onto a video monitor a selecteed section of said video signal corresponding to a 
selected one of said locations; 

tpositioning at each location one of a group of audio detectors; 
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arranging each said audio detector in operable combination with said camera means to 
activate said video signal control to project onto said video monitor an image of said 
location when one of said conferees located at said location generates a sound received by 
said respective audio detector. 
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ABSTRACT 



A surveillance system featuring a camera that is directed to focus on a location where a 
noise has been generated and display an image of the location. 1. The system includes an 
omnidirectional camera having a lens for viewing a 360° field of view,.a visual memory 
for storing visual data at addresses representing the field of view, an omnidirectional audial 
detector for detecting the dkection of location of a source of a signal, a display selection 
means for selecting visual data from one of said addresses m said visual memory 
corresponding to said direction of location and a monitor for displaying said selected 
visual data. In one embodiment, the audial detector is a plurality of detectors arranged on a 
circle. A register stores the address of the detector closest to the source and this address is 
transmitted to the display controller which conditions a system monitor to display an image 
representing the selected visual data. 
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